{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tf.layer 的API使用\n",
    "\n",
    "在前面的全部例子中使用的都是 TensorFlow 原生的 API。\n",
    "\n",
    "好处就是很多东西都要自己写，自己实现，自己对底层的实现也会更加了解。比如写一个全链接层，你需要自己创建权值矩阵 Weight 和 偏置向量 biases，包括它们使用什么初始化方式你都自己写。这样子你对于每一层的参数的维度都会非常清楚。\n",
    "\n",
    "缺点就是太麻烦了。\n",
    "\n",
    "这里来探索一下 TensorFlow 中的高级 API， tf.layer 对应的接口。主要是一些常用层的用法：\n",
    "- 全连接层\n",
    "- 卷积层\n",
    "- 转置卷积（反卷积）\n",
    "- BN 层\n",
    "\n",
    "相关的例子参考： https://github.com/aymericdamien/TensorFlow-Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # 不打印 warning \n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# 设置GPU按需增长\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "sess = tf.Session(config=config)\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.各种层的用法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 各种初始化方式 参考： http://www.cnblogs.com/denny402/p/6932956.html\n",
    "\n",
    "- 对于 biases 一般使用 常数初始化\n",
    "- 对于 kernel 一般会使用高斯初始化或者 Xavier 初始化的方式\n",
    "\n",
    "还有其他一些初始化的方式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 常用的 kernel 和 biases 的各种初始化方式\n",
    "constant_init = tf.constant_initializer(dtype=tf.float32, value=0.01)  # 初始化为常数，biases 常用\n",
    "zeros_init = tf.zeros_initializer(dtype=tf.float32)\n",
    "ones_init = tf.ones_initializer(dtype=tf.float32)\n",
    "\n",
    "truncated_normal_init = tf.truncated_normal_initializer(dtype=tf.float32, mean=0, stddev=0.01) # 截断高斯初始化,通常设置 stddev 就够了\n",
    "\n",
    "uniform_init = tf.random_uniform_initializer(minval=0.0, maxval=1.0, seed=None)  # 均匀初始化\n",
    "\n",
    "xavier_init = tf.contrib.layers.xavier_initializer(uniform=True)  # xavier 初始化，若 uniform=False,使用 normal distributed random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 全连接层 tf.layers.dense\n",
    "```python\n",
    "Signature: tf.layers.dense(inputs, units, activation=None, use_bias=True, kernel_initializer=None, bias_initializer=<tensorflow.python.ops.init_ops.Zeros object at 0x7fad69bae908>, kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, trainable=True, name=None, reuse=None)\n",
    "Docstring:\n",
    "Functional interface for the densely-connected layer.\n",
    "\n",
    "主要参数\n",
    "units: Integer or Long, dimensionality of the output space. 输出的维度\n",
    "activation: Activation function (callable). Set it to None to maintain a\n",
    "    linear activation. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable_3:0' shape=(100, 784) dtype=float32_ref>\n",
      "X_input.shape=[100, 784]\n",
      "Tensor(\"dense_1/Relu:0\", shape=(100, 1024), dtype=float32)\n",
      "fc1.shape=[100, 1024]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "in_dim = 784\n",
    "X_input = tf.Variable(tf.truncated_normal(dtype=tf.float32, shape=[batch_size, in_dim]))\n",
    "print(X_input)\n",
    "print('X_input.shape={}'.format(X_input.get_shape().as_list()))  # 获取 tensor 的维度\n",
    "\n",
    "fc1 = tf.layers.dense(inputs=X_input, units=1024, activation=tf.nn.relu, kernel_initializer=xavier_init, bias_initializer=constant_init)\n",
    "print(fc1)\n",
    "print('fc1.shape={}'.format(fc1.get_shape().as_list()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其实 tf.layers.dense 的接口中提供了非常丰富的参数设置，包括是否使用激活函数，使用什么激活函数，初始化方式，正则化方式等等。\n",
    "\n",
    "这样子我们就没有必要再自己花费大量的功夫来写自己的全连接层了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 卷积层 tf.layers.conv2d\n",
    "```python\n",
    "Signature: tf.layers.conv2d(inputs, filters, kernel_size, strides=(1, 1), padding='valid', data_format='channels_last', dilation_rate=(1, 1), activation=None, use_bias=True, kernel_initializer=None, bias_initializer=<tensorflow.python.ops.init_ops.Zeros object at 0x7fad69bf86d8>, kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, trainable=True, name=None, reuse=None)\n",
    "Docstring:\n",
    "Functional interface for the 2D convolution layer.\n",
    "\n",
    "主要的参数：\n",
    "Arguments: \n",
    "  inputs: Tensor input. 这里的 input 应该是一个 [batch_size, img_height, img_width, n_channel] 的向量\n",
    "  filters: 这一层中卷积核（输出）的深度。Integer, the dimensionality of the output space \n",
    "  kernel_size: 卷积核的大小，如 3 或者 (3, 3)\n",
    "  strides: 滑动步长，如（2,2）表示横向和纵向的滑动步长都是 2.\n",
    "  padding: One of `\"valid\"` or `\"same\"` (case-insensitive).\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable_10:0' shape=(100, 256, 256, 3) dtype=float32_ref>\n",
      "X_input.shape=[100, 256, 256, 3]\n",
      "conv1: Tensor(\"conv2d_8/BiasAdd:0\", shape=(100, 128, 128, 64), dtype=float32)\n",
      "conv2: Tensor(\"conv2d_9/BiasAdd:0\", shape=(100, 64, 64, 128), dtype=float32)\n",
      "conv3: Tensor(\"conv2d_10/BiasAdd:0\", shape=(100, 32, 32, 196), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "img_height = img_width = 256\n",
    "n_channel = 3\n",
    "X_input = tf.Variable(tf.truncated_normal(shape=[batch_size, img_height, img_width, n_channel]))\n",
    "print(X_input)\n",
    "print('X_input.shape={}'.format(X_input.get_shape().as_list()))\n",
    "\n",
    "# 卷积层的参数，这里写 3 个卷积层，方便后面转置卷积层\n",
    "conv_parms = {\n",
    "    'depth1': 64,       # kernel 深度\n",
    "    'k_size1': (5, 5),  # 卷积核大小\n",
    "    'stride1': (2, 2),  # 滑动步长\n",
    "    'padding1': 'same', # padding 方式\n",
    "    \n",
    "    'depth2': 128,      \n",
    "    'k_size2': (3, 3), \n",
    "    'stride2': (2, 2),  \n",
    "    'padding2': 'same', \n",
    "    \n",
    "    'depth3': 196,      \n",
    "    'k_size3': (3, 3), \n",
    "    'stride3': (2, 2),  \n",
    "    'padding3': 'same'\n",
    "}\n",
    "\n",
    "conv1 = tf.layers.conv2d(X_input, filters=conv_parms['depth1'], kernel_size=conv_parms['k_size1'], \n",
    "                         strides=conv_parms['stride1'], padding=conv_parms['padding1'])\n",
    "print('conv1:', conv1)\n",
    "conv2 = tf.layers.conv2d(conv1, filters=conv_parms['depth2'], kernel_size=conv_parms['k_size2'], \n",
    "                         strides=conv_parms['stride2'], padding=conv_parms['padding2'])\n",
    "print('conv2:', conv2)\n",
    "conv3 = tf.layers.conv2d(conv2, filters=conv_parms['depth3'], kernel_size=conv_parms['k_size3'], \n",
    "                         strides=conv_parms['stride3'], padding=conv_parms['padding3'])\n",
    "print('conv3:', conv3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 转置卷积 \n",
    "\n",
    "```python\n",
    "Signature: tf.layers.conv2d_transpose(inputs, filters, kernel_size, strides=(1, 1), padding='valid', data_format='channels_last', activation=None, use_bias=True, kernel_initializer=None, bias_initializer=<tensorflow.python.ops.init_ops.Zeros object at 0x7fad69bf8c88>, kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, trainable=True, name=None, reuse=None)\n",
    "Docstring:\n",
    "Functional interface for transposed 2D convolution layer.\n",
    "\n",
    "主要参数：和 conv2 的参数基本上是一样的，通过下面的例子就很好理解了。\n",
    "Arguments:\n",
    "  inputs: Input tensor.和 conv2d 一样，输入也是一个 [batch_size, img_height, img_width, n_channel] 的向量\n",
    "  filters: 转置卷积输出的卷积核深度。\n",
    "  kernel_size: 卷积核的大小。\n",
    "  strides: 滑动步长。\n",
    "  padding: one of `\"valid\"` or `\"same\"` (case-insensitive).\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tr_conv_input: Tensor(\"conv2d_10/BiasAdd:0\", shape=(100, 32, 32, 196), dtype=float32)\n",
      "tr_conv3: Tensor(\"conv2d_transpose_4/BiasAdd:0\", shape=(100, 64, 64, 128), dtype=float32)\n",
      "tr_conv2: Tensor(\"conv2d_transpose_5/BiasAdd:0\", shape=(100, 128, 128, 64), dtype=float32)\n",
      "tr_conv1 Tensor(\"conv2d_transpose_6/BiasAdd:0\", shape=(100, 256, 256, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "\"\"\" 把上面卷积部分的输出作为 transpoed convolution layer 的输入,\n",
    "下面通过转置卷积来还原原来的图片大小。在 DCGAN 的 generator 就是这样的一个结构。\n",
    "上面我们一共用了 3 层 conv2d，对应的这里我们应该使用三次反卷积\n",
    "\"\"\"\n",
    "tr_conv_input = conv3   \n",
    "print('tr_conv_input:', tr_conv_input)\n",
    "tr_conv3 = tf.layers.conv2d_transpose(tr_conv_input, filters=conv_parms['depth2'], kernel_size=conv_parms['k_size3'], \n",
    "                         strides=conv_parms['stride3'], padding=conv_parms['padding3'])\n",
    "print('tr_conv3:', tr_conv3)  # 这里得到的维度和 conv3 的输入应该一样，也就是和 conv2 的输出应该一样\n",
    "tr_conv2 = tf.layers.conv2d_transpose(tr_conv3, filters=conv_parms['depth1'], kernel_size=conv_parms['k_size2'], \n",
    "                         strides=conv_parms['stride2'], padding=conv_parms['padding2'])\n",
    "print('tr_conv2:', tr_conv2) \n",
    "tr_conv1 = tf.layers.conv2d_transpose(tr_conv2, filters=n_channel, kernel_size=conv_parms['k_size1'], \n",
    "                         strides=conv_parms['stride1'], padding=conv_parms['padding1'])\n",
    "print('tr_conv1', tr_conv1)  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的结果看起来很好理解吧，至于 conv2d_trasposed 最好使用原生的 API 写一下，这时候你才能比较好的理解参数的维度。可以参考：https://github.com/carpedm20/DCGAN-tensorflow/blob/master/ops.py\n",
    "\n",
    "但是实际上还是有些问题需要注意的，我们知道在 conv 中，当使用 padding 方式为 'same' 的时候，计算输出维度的时候我们有个上取整的操作。这也就是说不通的输入维度可能会有一样的输出维度，这个时候想要一模一样的回复到原来的维度就不行了。所以在设置图像大小的时候，每次 conv 输出的维度和下一层的 stride 都应该是整除的。\n",
    "\n",
    "比如下面就是一个恢复不了原图片大小的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable_13:0' shape=(100, 256, 256, 3) dtype=float32_ref>\n",
      "X_input.shape=[100, 256, 256, 3]\n",
      "conv1: Tensor(\"conv2d_17/BiasAdd:0\", shape=(100, 86, 86, 64), dtype=float32)\n",
      "conv2: Tensor(\"conv2d_18/BiasAdd:0\", shape=(100, 43, 43, 128), dtype=float32)\n",
      "conv3: Tensor(\"conv2d_19/BiasAdd:0\", shape=(100, 22, 22, 196), dtype=float32)\n",
      "tr_conv_input: Tensor(\"conv2d_19/BiasAdd:0\", shape=(100, 22, 22, 196), dtype=float32)\n",
      "tr_conv3: Tensor(\"conv2d_transpose_13/BiasAdd:0\", shape=(100, 44, 44, 128), dtype=float32)\n",
      "tr_conv2: Tensor(\"conv2d_transpose_14/BiasAdd:0\", shape=(100, 88, 88, 64), dtype=float32)\n",
      "tr_conv1 Tensor(\"conv2d_transpose_15/BiasAdd:0\", shape=(100, 264, 264, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "img_height = img_width = 256\n",
    "n_channel = 3\n",
    "X_input = tf.Variable(tf.truncated_normal(shape=[batch_size, img_height, img_width, n_channel]))\n",
    "print(X_input)\n",
    "print('X_input.shape={}'.format(X_input.get_shape().as_list()))\n",
    "\n",
    "conv_parms = {\n",
    "    'depth1': 64,       \n",
    "    'k_size1': (5, 5),  \n",
    "    'stride1': (3, 3),  # 滑动步长     ### 我只改了这里 ###\n",
    "    'padding1': 'same', \n",
    "    \n",
    "    'depth2': 128,      \n",
    "    'k_size2': (3, 3), \n",
    "    'stride2': (2, 2),  \n",
    "    'padding2': 'same', \n",
    "    \n",
    "    'depth3': 196,      \n",
    "    'k_size3': (3, 3), \n",
    "    'stride3': (2, 2),  \n",
    "    'padding3': 'same'\n",
    "}\n",
    "\n",
    "conv1 = tf.layers.conv2d(X_input, filters=conv_parms['depth1'], kernel_size=conv_parms['k_size1'], \n",
    "                         strides=conv_parms['stride1'], padding=conv_parms['padding1'])\n",
    "print('conv1:', conv1)\n",
    "conv2 = tf.layers.conv2d(conv1, filters=conv_parms['depth2'], kernel_size=conv_parms['k_size2'], \n",
    "                         strides=conv_parms['stride2'], padding=conv_parms['padding2'])\n",
    "print('conv2:', conv2)\n",
    "conv3 = tf.layers.conv2d(conv2, filters=conv_parms['depth3'], kernel_size=conv_parms['k_size3'], \n",
    "                         strides=conv_parms['stride3'], padding=conv_parms['padding3'])\n",
    "print('conv3:', conv3)\n",
    "\n",
    "\n",
    "tr_conv_input = conv3   \n",
    "print('tr_conv_input:', tr_conv_input)\n",
    "tr_conv3 = tf.layers.conv2d_transpose(tr_conv_input, filters=conv_parms['depth2'], kernel_size=conv_parms['k_size3'], \n",
    "                         strides=conv_parms['stride3'], padding=conv_parms['padding3'])\n",
    "print('tr_conv3:', tr_conv3)  \n",
    "tr_conv2 = tf.layers.conv2d_transpose(tr_conv3, filters=conv_parms['depth1'], kernel_size=conv_parms['k_size2'], \n",
    "                         strides=conv_parms['stride2'], padding=conv_parms['padding2'])\n",
    "print('tr_conv2:', tr_conv2) \n",
    "tr_conv1 = tf.layers.conv2d_transpose(tr_conv2, filters=n_channel, kernel_size=conv_parms['k_size1'], \n",
    "                         strides=conv_parms['stride1'], padding=conv_parms['padding1'])\n",
    "print('tr_conv1', tr_conv1)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tf.layers.dropout\n",
    "```python\n",
    "ignature: tf.layers.dropout(inputs, rate=0.5, noise_shape=None, seed=None, training=False, name=None)\n",
    "Docstring:\n",
    "Applies Dropout to the input.\n",
    "\n",
    "需要注意这里的 rate 是丢弃的 rate，也就是 rate 应该等于 1.0 - keep_prob\n",
    "Arguments:\n",
    "  inputs: Tensor input.\n",
    "  rate: The dropout rate, between 0 and 1. E.g. \"rate=0.1\" would drop out 10% of input units.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tr_conv1_drop: Tensor(\"dropout_4/Identity:0\", shape=(100, 264, 264, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tr_conv1_drop = tf.layers.dropout(tr_conv1, rate=0.2)\n",
    "print('tr_conv1_drop:', tr_conv1_drop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tf.layers.batch_normalization\n",
    "\n",
    "关于 batch_normalization 经常会出问题，主要是训练的时候用的是 mini-batch 的均值和方差，而在测试的时候使用的指数平均的均值方差。\n",
    "\n",
    "理解可以参考： [tensorflow中batch normalization的用法](https://www.cnblogs.com/hrlnw/p/7227447.html)\n",
    "\n",
    "\n",
    "```python\n",
    "Signature: tf.layers.batch_normalization(inputs, axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True, beta_initializer=<tensorflow.python.ops.init_ops.Zeros object at 0x7fad69bce710>, gamma_initializer=<tensorflow.python.ops.init_ops.Ones object at 0x7fad69bce748>, moving_mean_initializer=<tensorflow.python.ops.init_ops.Zeros object at 0x7fad69bce780>, moving_variance_initializer=<tensorflow.python.ops.init_ops.Ones object at 0x7fad69bce7b8>, beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None, training=False, trainable=True, name=None, reuse=None, renorm=False, renorm_clipping=None, renorm_momentum=0.99, fused=None, virtual_batch_size=None, adjustment=None)\n",
    "Docstring:\n",
    "Functional interface for the batch normalization layer.\n",
    "\n",
    "Reference: http://arxiv.org/abs/1502.03167\n",
    "\n",
    "\"Batch Normalization: Accelerating Deep Network Training by Reducing\n",
    "Internal Covariate Shift\"\n",
    "\n",
    "Sergey Ioffe, Christian Szegedy\n",
    "\n",
    "Note: when training, the moving_mean and moving_variance need to be updated.\n",
    "By default the update ops are placed in `tf.GraphKeys.UPDATE_OPS`, so they\n",
    "need to be added as a dependency to the `train_op`. For example:\n",
    "```\n",
    "```python\n",
    "  update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "  with tf.control_dependencies(update_ops):  # 这句话的意思是当运行下面的内容(train_op) 时，一定先执行 update_ops 的所有操作\n",
    "    train_op = optimizer.minimize(loss)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 举个简单的 MNIST 分类的例子\n",
    "参考：https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:219: retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use urllib or similar directly.\n",
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n",
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmps1gdlnuw\n",
      "INFO:tensorflow:Using config: {'_global_id_in_cluster': 0, '_task_id': 0, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_save_summary_steps': 100, '_session_config': None, '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_is_chief': True, '_num_worker_replicas': 1, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9ae7a24668>, '_master': '', '_evaluation_master': '', '_tf_random_seed': None, '_model_dir': '/tmp/tmps1gdlnuw', '_save_checkpoints_steps': None, '_task_type': 'worker', '_service': None, '_log_step_count_steps': 100}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /tmp/tmps1gdlnuw/model.ckpt.\n",
      "INFO:tensorflow:loss = 2.308074, step = 0\n",
      "INFO:tensorflow:global_step/sec: 188.298\n",
      "INFO:tensorflow:loss = 0.06075897, step = 100 (0.533 sec)\n",
      "INFO:tensorflow:global_step/sec: 258.738\n",
      "INFO:tensorflow:loss = 0.14094348, step = 200 (0.390 sec)\n",
      "INFO:tensorflow:global_step/sec: 233.012\n",
      "INFO:tensorflow:loss = 0.059998624, step = 300 (0.428 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.72\n",
      "INFO:tensorflow:loss = 0.0284447, step = 400 (0.486 sec)\n",
      "INFO:tensorflow:global_step/sec: 281.684\n",
      "INFO:tensorflow:loss = 0.02473668, step = 500 (0.351 sec)\n",
      "INFO:tensorflow:global_step/sec: 185.211\n",
      "INFO:tensorflow:loss = 0.1192817, step = 600 (0.543 sec)\n",
      "INFO:tensorflow:global_step/sec: 212.076\n",
      "INFO:tensorflow:loss = 0.049546495, step = 700 (0.468 sec)\n",
      "INFO:tensorflow:global_step/sec: 258.789\n",
      "INFO:tensorflow:loss = 0.07456283, step = 800 (0.389 sec)\n",
      "INFO:tensorflow:global_step/sec: 233.956\n",
      "INFO:tensorflow:loss = 0.029562324, step = 900 (0.428 sec)\n",
      "INFO:tensorflow:global_step/sec: 219.345\n",
      "INFO:tensorflow:loss = 0.07197396, step = 1000 (0.454 sec)\n",
      "INFO:tensorflow:global_step/sec: 228.833\n",
      "INFO:tensorflow:loss = 0.03840988, step = 1100 (0.436 sec)\n",
      "INFO:tensorflow:global_step/sec: 236.806\n",
      "INFO:tensorflow:loss = 0.021077603, step = 1200 (0.423 sec)\n",
      "INFO:tensorflow:global_step/sec: 228.135\n",
      "INFO:tensorflow:loss = 0.050501708, step = 1300 (0.443 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.927\n",
      "INFO:tensorflow:loss = 0.041159876, step = 1400 (0.478 sec)\n",
      "INFO:tensorflow:global_step/sec: 221.961\n",
      "INFO:tensorflow:loss = 0.008115782, step = 1500 (0.455 sec)\n",
      "INFO:tensorflow:global_step/sec: 197.505\n",
      "INFO:tensorflow:loss = 0.08929724, step = 1600 (0.502 sec)\n",
      "INFO:tensorflow:global_step/sec: 229.869\n",
      "INFO:tensorflow:loss = 0.0006037804, step = 1700 (0.437 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.817\n",
      "INFO:tensorflow:loss = 0.020877827, step = 1800 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 226.803\n",
      "INFO:tensorflow:loss = 0.013362966, step = 1900 (0.439 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/tmps1gdlnuw/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.012543799.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-05-17-13:01:21\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmps1gdlnuw/model.ckpt-2000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2018-05-17-13:01:22\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.9911, global_step = 2000, loss = 0.028943231\n",
      "Testing Accuracy: 0.9911\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Convolutional Neural Network.\n",
    "Author: Aymeric Damien\n",
    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
    "\"\"\"\n",
    "from __future__ import division, print_function, absolute_import\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"../data/MNIST_data/\", one_hot=False)\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Training Parameters\n",
    "learning_rate = 0.001\n",
    "num_steps = 2000\n",
    "batch_size = 128\n",
    "\n",
    "# Network Parameters\n",
    "num_input = 784 # MNIST data input (img shape: 28*28)\n",
    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
    "dropout = 0.25 # Dropout, probability to drop a unit\n",
    "\n",
    "\n",
    "# Create the neural network\n",
    "def conv_net(x_dict, n_classes, dropout, reuse, is_training):\n",
    "    # Define a scope for reusing the variables\n",
    "    with tf.variable_scope('ConvNet', reuse=reuse):\n",
    "        # TF Estimator input is a dict, in case of multiple inputs\n",
    "        x = x_dict['images']\n",
    "\n",
    "        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n",
    "        # Reshape to match picture format [Height x Width x Channel]\n",
    "        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n",
    "        x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
    "\n",
    "        # Convolution Layer with 32 filters and a kernel size of 5\n",
    "        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)\n",
    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
    "        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)\n",
    "\n",
    "        # Convolution Layer with 64 filters and a kernel size of 3\n",
    "        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)\n",
    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
    "        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)\n",
    "\n",
    "        # Flatten the data to a 1-D vector for the fully connected layer\n",
    "        fc1 = tf.contrib.layers.flatten(conv2)\n",
    "\n",
    "        # Fully connected layer (in tf contrib folder for now)\n",
    "        fc1 = tf.layers.dense(fc1, 1024)\n",
    "        # Apply Dropout (if is_training is False, dropout is not applied)\n",
    "        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)\n",
    "\n",
    "        # Output layer, class prediction\n",
    "        out = tf.layers.dense(fc1, n_classes)\n",
    "\n",
    "    return out\n",
    "\n",
    "\n",
    "# Define the model function (following TF Estimator Template)\n",
    "def model_fn(features, labels, mode):\n",
    "    # Build the neural network\n",
    "    # Because Dropout have different behavior at training and prediction time, we\n",
    "    # need to create 2 distinct computation graphs that still share the same weights.\n",
    "    logits_train = conv_net(features, num_classes, dropout, reuse=False,\n",
    "                            is_training=True)\n",
    "    logits_test = conv_net(features, num_classes, dropout, reuse=True,\n",
    "                           is_training=False)\n",
    "\n",
    "    # Predictions\n",
    "    pred_classes = tf.argmax(logits_test, axis=1)\n",
    "    pred_probas = tf.nn.softmax(logits_test)\n",
    "\n",
    "    # If prediction mode, early return\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)\n",
    "\n",
    "        # Define loss and optimizer\n",
    "    loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
    "        logits=logits_train, labels=tf.cast(labels, dtype=tf.int32)))\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "    train_op = optimizer.minimize(loss_op,\n",
    "                                  global_step=tf.train.get_global_step())\n",
    "\n",
    "    # Evaluate the accuracy of the model\n",
    "    acc_op = tf.metrics.accuracy(labels=labels, predictions=pred_classes)\n",
    "\n",
    "    # TF Estimators requires to return a EstimatorSpec, that specify\n",
    "    # the different ops for training, evaluating, ...\n",
    "    estim_specs = tf.estimator.EstimatorSpec(\n",
    "        mode=mode,\n",
    "        predictions=pred_classes,\n",
    "        loss=loss_op,\n",
    "        train_op=train_op,\n",
    "        eval_metric_ops={'accuracy': acc_op})\n",
    "\n",
    "    return estim_specs\n",
    "\n",
    "\n",
    "# Build the Estimator\n",
    "model = tf.estimator.Estimator(model_fn)\n",
    "\n",
    "# Define the input function for training\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={'images': mnist.train.images}, y=mnist.train.labels,\n",
    "    batch_size=batch_size, num_epochs=None, shuffle=True)\n",
    "# Train the Model\n",
    "model.train(input_fn, steps=num_steps)\n",
    "\n",
    "# Evaluate the Model\n",
    "# Define the input function for evaluating\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={'images': mnist.test.images}, y=mnist.test.labels,\n",
    "    batch_size=batch_size, shuffle=False)\n",
    "# Use the Estimator 'evaluate' method\n",
    "e = model.evaluate(input_fn)\n",
    "\n",
    "print(\"Testing Accuracy:\", e['accuracy'])"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
